Introduction

What is this

This notebook will include informal meta-analyses of different metrics and methods for evaluating surgical skill.

The reported metrics compare differences between novices and expert surgeons.

It is informal because it’s not based on systematic review, and because some studies have been included with very relaxed conditions. For example, I have picked the novices and experts without comparing their definitions between studies. Novice = weakest skill group in the study, expert = strongest skill group in the study. If a study included more than 2 groups, I picked the weakest (=novice) and strongest (=expert) groups’ results and discarded the others. If a study included more than 1 task, or several sub-tasks, I picked the one with largest difference between groups.

Many papers did report means and standard deviations explicitly, so they had to be estimated from boxplots/barplots, or by some other means

For example, sometimes studies reported only mean or median, but no SE/SD. I estimated the SD/SE in those cases based e.g. on the SD of some other similar metric that they reported, or the SD of previous results for the same metric. See the excel file for notes on each study.

May or may not be turned into more systematic meta-analysis later.

Example metrics that will be most likely included (Bolded ones have priority)

Task time
Tool Path length
Tool Velocity
Tool Acceleration
Tool Curvature
Idle time
Pupil dilations
Blinks
Tool Movement efficiency
Number of movements
Tool Forces
Tool Torques
Bimanual dexterity
Jerk
Fixation duration
Saccade amplitudes
EEG?

Full list of papers and metrics can be found in the excel file shared in the repo:

Link to Github repo

Last update: 20.6.2022.

Submit results

If you notice errors or know some good studies to be included, feel free to forward them to

jani.koskinen [ at ] uef.fi

or use the form below TBD

How results are calculated

From each study, extract

Number of trials per group (Nn, Ne, for novices and experts, respectively)
Means per group (Mn, Me for novices and experts, respectively)
Standard deviations per group (SDn, SDe)

Calculate pooled standard deviation SDpooled
Normalize by calculating Standardized Mean Difference (SDM): (Mn - Me)/SDpooled
Calculate small sample size correction g = SMD*(1 - 3/(4n - 9)), where n is the total sample size of the study (both groups combined).
Calculate SDg, standard deviation after correction

These values are used as input in the R meta package’s metagen function.

For more information, check:

Doing Meta-Analysis with R: A Hands-On Guide

Forest plots

Forest plot explanation

Sample size estimation

How many samples needed at some effect size d? At alpha = 0.05 and power = 0.8 and using t-test. Assuming independent trials (e.g. no multiple measurements from same participants etc.)

Hover mouse over the points in the plot to see the values. Some baseline effect sizes from the meta-analyses given as baseline:

IT = Idle Time

TT = Task Time

BD = Bimanual Dexterity

TEPR = Task-Evoked Pupil Reaction/Dilation (Esimated without one outlier study removed)

TJ = Tool Jerk

TF = Tool Forceg

Task Time

Task time is the time taken to complete a task. Task can be short like a single knot or some longer complex task.

Studies

Load data

df.time <- read_excel('data/surgical_metrics.xlsx', sheet='task_time')

Print studies

Author	Year	Study	Journal
Koskinen et al.	2022	Utilizing Grasp Monitoring to Predict Microsurgical Expertise	Journal of Surgical Research
Chainey et al.	2021	Eye-Hand Coordination of Neurosurgeons: Evidence of Action-Related Fixation in Microsuturing	World Neurosurgery
Harada et al.	2015	Assessing microneurosurgical skill with medico-engineering technology	World Neurosurgery
Vedula et al.	2016	Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment	Journal of Surgical Education
Judkins et al.	2009	Objective evaluation of expert and novice performance during robotic surgical training tasks	Surgical Endoscopy
Smith et al.	2002	Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy	Surgical Endoscopy and Other Interventional Techniques
Francis et al.	2002	The performance of master surgeons on the Advanced Dundee Endoscopic Psychomotor Tester: Contrast validity study	Archives of Surgery
Moorthy et al.	2004	Bimodal assessment of laparoscopic suturing skills: Construct and concurrent validity	Surgical Endoscopy and Other Interventional Techniques
Van Sickle et al.	2008	Construct validity of an objective assessment method for laparoscopic intracorporeal suturing and knot tying	The American Journal of Surgery
Xeroulis et al.	2009	Simulation in laparoscopic surgery: A concurrent validity study for FLS	Surgical Endoscopy and Other Interventional Techniques
Huffman et al.	2020	Optimizing Assessment of Surgical Knot Tying Skill	Journal of Surgical Education
Law et al.	2004	Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment	Proceedings of the Eye tracking research & applications symposium on Eye tracking research & applications - ETRA’2004
Kazemi et al.	2010	Assessing suturing techniques using a virtual reality surgical simulator	Microsurgery
O’Toole et al.	1999	Measuring and Developing Suturing Technique with a Virtual Reality Surgical Simulator	Journal ofthe American College of Surgeons
Zheng et al.	2021	Action-related eye measures to assess surgical expertise	BJS Open
Datta et al.	2001	The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model	Journal of the American College of Surgeons
Pagador et al.	2012	Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment	International Journal of Computer Assisted Radiology and Surgery
Aggarwal et al.	2007	An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room	Annals of Surgery
Wilson et al.	2010	Psychomotor control in a virtual laparoscopic surgery training environment: Gaze control parameters differentiate novices from experts	Surgical Endoscopy
Hofstad et al.	2013	A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance	Surgical Endoscopy and Other Interventional Techniques
Hung et al.	2018	Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study	Journal of Urology
Yamaguchi et al.	2011	Objective assessment of laparoscopic suturing skills using a motion-tracking system	Surgical Endoscopy
Pellen et al.	2009	Laparoscopic surgical skills assessment: Can simulators replace experts?	World Journal of Surgery
Pastewski et al.	2021	Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills	Journal of Surgical Education
Chmarra et al.	2010	Objective classification of residents based on their psychomotor laparoscopic skills	Surgical Endoscopy and Other Interventional Techniques

Results

m.time <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.time,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Time to completion in Surgery")

summary(m.time)

## Review:     Time to completion in Surgery
## 
##                      SMD             95%-CI %W(random)
## Koskinen et al.   1.8413 [ 1.4135;  2.2691]        4.5
## Chainey et al.    0.7034 [ 0.0383;  1.3686]        4.4
## Harada et al.     1.5503 [ 0.8551;  2.2456]        4.4
## Vedula et al.     2.2149 [ 1.7299;  2.6999]        4.5
## Judkins et al.    5.3971 [ 3.8216;  6.9726]        3.7
## Smith et al.      8.0559 [ 5.5551; 10.5568]        2.9
## Francis et al.    0.9801 [ 0.3227;  1.6375]        4.4
## Moorthy et al.    1.4157 [ 0.3397;  2.4917]        4.1
## Van Sickle et al. 2.1365 [ 1.0202;  3.2528]        4.1
## Xeroulis et al.   2.5525 [ 1.2650;  3.8400]        4.0
## Huffman et al.    6.5116 [ 4.9174;  8.1059]        3.7
## Law et al.        2.0257 [ 1.3401;  2.7112]        4.4
## Kazemi et al.     0.8354 [-0.3084;  1.9791]        4.1
## O'Toole et al.    1.7086 [ 0.6569;  2.7602]        4.1
## Zheng et al.      1.9382 [ 0.6894;  3.1871]        4.0
## Datta et al.      2.1791 [ 1.1762;  3.1819]        4.2
## Pagador et al.    6.3695 [ 2.5221; 10.2170]        2.0
## Aggarwal et al.   0.1873 [-0.4390;  0.8136]        4.4
## Wilson et al.     1.3349 [ 0.1520;  2.5179]        4.1
## Hofstad et al.    1.3791 [ 0.3199;  2.4382]        4.1
## Hung et al.       2.2342 [ 1.7203;  2.7481]        4.5
## Yamaguchi et al.  4.5870 [ 2.7625;  6.4116]        3.5
## Pellen et al.     5.6362 [ 3.6128;  7.6596]        3.3
## Pastewski et al.  0.5600 [-0.1171;  1.2371]        4.4
## Chmarra et al.    0.9810 [ 0.0706;  1.8914]        4.2
## 
## Number of studies combined: k = 25
## 
##                         SMD           95%-CI    t  p-value
## Random effects model 2.3748 [1.5627; 3.1869] 6.04 < 0.0001
## 
## Quantifying heterogeneity:
##  tau^2 = 2.9577 [1.8111; 7.7816]; tau = 1.7198 [1.3458; 2.7895]
##  I^2 = 86.4% [81.1%; 90.2%]; H = 2.71 [2.30; 3.19]
## 
## Test of heterogeneity:
##       Q d.f.  p-value
##  176.04   24 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.time,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Time to completion in Surgery")

#dev.print(pdf, "figures/forest_time.pdf", width=10, height=10)

Discussion

Time to completion is by far the most often reported metric. It is often reported even when it is not the main focus of the study.

Bimanual Dexterity

Bimanual dexterity is a measure of how well the surgeon is able to use both hands at the same time. Note that there are many different ways for calculating “ability to use both hands simultaneously.”

Studies

Load data

df.biman <- read_excel('data/surgical_metrics.xlsx', sheet='tool_bimanual')

Print studies

Author	Year	Study	Journal
Koskinen et al.	2022	Movement-level process modeling of microsurgical bimanual and unimanual tasks	International Journal of Computer Assisted Radiology and Surgery
Hofstad et al.	2017	Psychomotor skills assessment by motion analysis in minimally invasive surgery on an animal organ	Minimally Invasive Therapy and Allied Technologies
Demirel et al.	2022	Scoring metrics for assessing skills in arthroscopic rotator cuff repair: performance comparison study of novice and expert surgeons	International Journal of Computer Assisted Radiology and Surgery
Islam et al.	2016	Affordable, web-based surgical skill training and evaluation tool	Journal of Biomedical Informatics
Zulbaran-Rojas et al.	2021	Utilization of Flexible-Wearable Sensors to Describe the Kinematics of Surgical Proficiency	Journal of Surgical Research
Mori et al.	2022	Validation of a novel virtual reality simulation system with the focus on training for surgical dissection during laparoscopic sigmoid colectomy	BMC Surgery

Results

Run meta-analysis

m.biman <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.biman,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Bimanual dexterity in Surgery")

Print results

summary(m.biman)

## Review:     Bimanual dexterity in Surgery
## 
##                           SMD              95%-CI %W(random)
## Koskinen et al.       -3.0589 [ -3.8825; -2.2353]       17.4
## Hofstad et al.        -3.0127 [ -4.6473; -1.3782]       16.0
## Demirel et al.        -2.0314 [ -3.0378; -1.0251]       17.1
## Islam et al.          -8.6969 [-10.7900; -6.6039]       15.1
## Zulbaran-Rojas et al. -0.8250 [ -1.7586;  0.1085]       17.2
## Mori et al.           -2.6867 [ -3.6936; -1.6799]       17.1
## 
## Number of studies combined: k = 6
## 
##                          SMD             95%-CI     t p-value
## Random effects model -3.2752 [-6.0448; -0.5057] -3.04  0.0288
## 
## Quantifying heterogeneity:
##  tau^2 = 6.0648 [2.0109; 43.9963]; tau = 2.4627 [1.4181; 6.6330]
##  I^2 = 89.8% [80.5%; 94.7%]; H = 3.13 [2.27; 4.33]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  49.11    5 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.biman,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Bimanual dexterity in Surgery")

#dev.print(pdf, "figures/forest_biman.pdf", width=8, height=8)

Discussion

Analysis of bimanual dexterity is made harder because there are so many different definitions for it.

Tool Movements

Number of tool movements made during the task. Note: I have included here the grasp results from our paper (and other studies that analyzed only one type of action/movement)

Studies

Load data

df.toolmvt <- read_excel('data/surgical_metrics.xlsx', sheet='tool_movements')

Print studies

Author	Year	Study	Journal
Datta et al.	2001	The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model	Journal of the American College of Surgeons
Pagador et al.	2012	Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment	International Journal of Computer Assisted Radiology and Surgery
Koskinen et al.	2022	Utilizing Grasp Monitoring to Predict Microsurgical Expertise	Journal of Surgical Research
Bann et al.	2003	Measurement of surgical dexterity using motion analysis of simple bench tasks	World Journal of Surgery
Smith et al.	2002	Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy	Surgical Endoscopy and Other Interventional Techniques
Aggarwal et al.	2007	An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room	Annals of Surgery
Yamaguchi et al.	2007	Construct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulator	Surgical Endoscopy and Other Interventional Techniques
Goldbraikh et al.	2021	Video-based fully automatic assessment of open surgery suturing skills	International Journal of Computer Assisted Radiology and Surgery
Vedula et al.	2016	Task-Level vs . Segment-Level Quantitative Metrics for Surgical Skill Assessment	Journal of Surgical Education
Wilson et al.	2010	Psychomotor control in a virtual laparoscopic surgery training environment: Gaze control parameters differentiate novices from experts	Surgical Endoscopy
Hofstad et al.	2013	A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance	Surgical Endoscopy and Other Interventional Techniques

Results

Run meta-analysis

m.toolmvt <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolmvt,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool movements in Surgery")

summary(m.toolmvt)

## Review:     Tool movements in Surgery
## 
##                       SMD             95%-CI %W(random)
## Datta et al.       2.0390 [ 1.0607;  3.0174]        9.6
## Pagador et al.    10.0866 [ 4.2364; 15.9368]        3.7
## Koskinen et al.    1.3393 [ 0.7781;  1.9006]       10.0
## Bann et al.        1.2504 [ 0.4629;  2.0380]        9.8
## Smith et al.       5.9403 [ 4.0136;  7.8671]        8.5
## Aggarwal et al.   -0.0641 [-0.6894;  0.5612]        9.9
## Yamaguchi et al.   2.3074 [ 1.3887;  3.2260]        9.7
## Goldbraikh et al.  2.6143 [ 1.8535;  3.3751]        9.8
## Vedula et al.      6.5233 [ 5.6418;  7.4047]        9.7
## Wilson et al.      0.5955 [-0.4889;  1.6799]        9.5
## Hofstad et al.     0.9586 [-0.0444;  1.9617]        9.6
## 
## Number of studies combined: k = 11
## 
##                         SMD           95%-CI    t p-value
## Random effects model 2.5911 [0.8299; 4.3523] 3.28  0.0083
## 
## Quantifying heterogeneity:
##  tau^2 = 5.1496 [2.4728; 26.0028]; tau = 2.2693 [1.5725; 5.0993]
##  I^2 = 94.7% [92.2%; 96.4%]; H = 4.33 [3.58; 5.24]
## 
## Test of heterogeneity:
##       Q d.f.  p-value
##  187.46   10 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolmvt,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool movements in Surgery")

#dev.print(pdf, "figures/forest_toolmvt.pdf", width=8, height=8)

Discussion

Tool movements are perhaps the second most often reported metric. Different papers measure, analyze and report them differently. Often connected to “movement efficiency”.

Tool Idle Time

Tool idle time measures how long the tools were not being used, either as time or as fraction of the complete task time.

Studies

Load data

df.toolidle <- read_excel('data/surgical_metrics.xlsx', sheet='tool_idle')

Print studies

Author	Year	Study	Journal
Koskinen et al.	2021	Movement-level process modeling of microsurgical bimanual and unimanual tasks	International Journal of Computer Assisted Radiology and Surgery
Uemura et al.	2015	Procedural surgical skill assessment in laparoscopic training environments	International Journal of Computer Assisted Radiology and Surgery
D’Angelo et al.	2015	Idle time: An underdeveloped performance metric for assessing surgical skill	American Journal of Surgery
Mackenzie et al.	2021	Enhanced Training Benefits of Video Recording Surgery With Automated Hand Motion Analysis	World Journal of Surgery
Oropesa et al.	2013	Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery	Surgical Innovation
Hung et al.	2018	Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study	Journal of Urology
Topalli et al.	2018	Eye-Hand Coordination Patterns of Intermediate and Novice Surgeons in a Simulation-Based Endoscopic Surgery Training Environment	Journal of Eye Movement Research

Results

Run meta-analysis

m.toolidle <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolidle,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Idle time in Surgery")

summary(m.toolidle)

## Review:     Idle time in Surgery
## 
##                     SMD            95%-CI %W(random)
## Koskinen et al.  2.5600 [ 1.8069; 3.3131]       17.7
## Uemura et al.    1.6933 [ 0.7816; 2.6050]       16.2
## D'Angelo et al.  2.7724 [ 1.0363; 4.5085]        9.3
## Mackenzie et al. 0.3642 [-1.6449; 2.3734]        7.7
## Oropesa et al.   0.9556 [-0.1828; 2.0941]       14.0
## Hung et al.      0.6990 [ 0.2837; 1.1144]       20.8
## Topalli et al.   0.6159 [-0.4828; 1.7147]       14.3
## 
## Number of studies combined: k = 7
## 
##                         SMD           95%-CI    t p-value
## Random effects model 1.3803 [0.5279; 2.2326] 3.96  0.0074
## 
## Quantifying heterogeneity:
##  tau^2 = 0.5520 [0.0915; 4.0820]; tau = 0.7430 [0.3024; 2.0204]
##  I^2 = 75.3% [47.7%; 88.3%]; H = 2.01 [1.38; 2.93]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  24.29    6  0.0005
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolidle,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Idle time in Surgery")

#dev.print(pdf, "figures/forest_toolidle.pdf", width=8, height=8)

Discussion

Not many papers that focused on idle time.

Tool Path Length

How much the tools travel during the task.

Studies

Load data

df.toolpl <- read_excel('data/surgical_metrics.xlsx', sheet='tool_path_length')

Print studies

Author	Year	Study	Journal
Aggarwal et al.	2007	An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room	Annals of Surgery
Moorthy et al.	2004	Bimodal assessment of laparoscopic suturing skills: Construct and concurrent validity	Surgical Endoscopy and Other Interventional Techniques
Smith et al.	2002	Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy	Surgical Endoscopy and Other Interventional Techniques
Pagador et al.	2012	Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment	International Journal of Computer Assisted Radiology and Surgery
Goldbraikh et al.	2021	Video-based fully automatic assessment of open surgery suturing skills	International Journal of Computer Assisted Radiology and Surgery
Jimbo et al.	2017	A new innovative laparoscopic fundoplication training simulator with a surgical skill validation system	Surgical Endoscopy
Hofstad et al.	2013	A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance	Surgical Endoscopy and Other Interventional Techniques
Oropesa et al.	2013	Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery	Surgical Innovation
Pellen et al.	2009	Laparoscopic surgical skills assessment: Can simulators replace experts?	World Journal of Surgery
D’Angelo et al.	2015	Idle time: An underdeveloped performance metric for assessing surgical skill	American Journal of Surgery
Hung et al.	2018	Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study	Journal of Urology
Vedula et al.	2016	Task-Level vs . Segment-Level Quantitative Metrics for Surgical Skill Assessment	Journal of Surgical Education
Yamaguchi et al.	2011	Objective assessment of laparoscopic suturing skills using a motion-tracking system	Surgical Endoscopy
Harada et al.	2015	Assessing Microneurosurgical Skill with Medico-Engineering Technology	World Neurosurgery
Ebina et al.	2021	Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs	Surgical Endoscopy
Chmarra et al.	2010	Objective classification of residents based on their psychomotor laparoscopic skills	Surgical Endoscopy and Other Interventional Techniques

Results

Run meta-analysis

m.toolpl <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolpl,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool path length in Surgery")

summary(m.toolpl)

## Review:     Tool path length in Surgery
## 
##                      SMD             95%-CI %W(random)
## Aggarwal et al.   0.0647 [-0.5606;  0.6900]        7.6
## Moorthy et al.    1.3161 [ 0.2542;  2.3780]        6.0
## Smith et al.      2.6541 [ 1.5194;  3.7889]        5.7
## Pagador et al.    6.2865 [ 2.4827; 10.0904]        1.2
## Goldbraikh et al. 2.0174 [ 1.3325;  2.7024]        7.4
## Jimbo et al.      0.8695 [ 0.1950;  1.5441]        7.5
## Hofstad et al.    0.7989 [-0.1876;  1.7853]        6.3
## Oropesa et al.    0.2889 [-0.8108;  1.3885]        5.8
## Pellen et al.     2.0960 [ 0.9877;  3.2042]        5.8
## D'Angelo et al.   1.7506 [ 0.3193;  3.1819]        4.7
## Hung et al.       1.8000 [ 1.3220;  2.2779]        8.1
## Vedula et al.     2.4204 [ 1.9215;  2.9194]        8.1
## Yamaguchi et al.  3.3661 [ 1.8874;  4.8449]        4.5
## Harada et al.     1.0214 [ 0.3743;  1.6685]        7.6
## Ebina et al.      0.9071 [ 0.1861;  1.6281]        7.3
## Chmarra et al.    1.2076 [ 0.2706;  2.1447]        6.4
## 
## Number of studies combined: k = 16
## 
##                         SMD           95%-CI    t  p-value
## Random effects model 1.5109 [0.9657; 2.0561] 5.91 < 0.0001
## 
## Quantifying heterogeneity:
##  tau^2 = 0.5801 [0.2782; 3.3711]; tau = 0.7617 [0.5275; 1.8360]
##  I^2 = 78.4% [65.5%; 86.5%]; H = 2.15 [1.70; 2.72]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  69.48   15 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolpl,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool path length in Surgery")

#dev.print(pdf, "figures/forest_toolpl.pdf", width=8, height=8)

Discussion

Tool path length also a very common metric. Most studies report that novices have much larger path length, indicating less effective movements. Results differ based on task and surgical

Tool Velocity

Tool velocity/speed measures how fast the surgical tool or tools are moving.

Studies

Load data

df.toolvelocity <- read_excel('data/surgical_metrics.xlsx', sheet='tool_velocity')

Print studies

Author	Year	Study	Journal
Davids et al.	2021	Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation.	World Neurosurgery
Pastewski et al.	2021	Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills	Journal of Surgical Education
Hwang et al.	2006	Correlating motor performance with surgical error in laparoscopic cholecystectomy	Surgical Endoscopy and Other Interventional Techniques
Ebina et al.	2021	Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs	Surgical Endoscopy
Jimbo et al.	2017	A new innovative laparoscopic fundoplication training simulator with a surgical skill validation system	Surgical Endoscopy
Judkins et al.	2009	Objective evaluation of expert and novice performance during robotic surgical training tasks	Surgical Endoscopy
Hofstad et al.	2013	A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance	Surgical Endoscopy and Other Interventional Techniques
Frasier et al.	2016	A marker-less technique for measuring kinematics in the operating room	Surgery (United States)
Azari et al.	2018	Can surgical performance for varying experience be measured from hand motions?	Proceedings of the Human Factors and Ergonomics Society
Pagador et al.	2012	Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment	International Journal of Computer Assisted Radiology and Surgery

Results

Run meta-analysis

m.toolvelocity <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolvelocity,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool velocity in Surgery")

summary(m.toolvelocity)

## Review:     Tool velocity in Surgery
## 
##                      SMD             95%-CI %W(random)
## Davids et al.     0.5140 [-1.5370;  2.5650]        5.0
## Pastewski et al. -0.7177 [-1.4028; -0.0326]       12.8
## Hwang et al.      6.1176 [ 1.5045; 10.7307]        1.3
## Ebina et al.     -0.8684 [-1.5865; -0.1503]       12.5
## Jimbo et al.     -0.7654 [-1.4334; -0.0974]       12.9
## Judkins et al.    0.6675 [-0.0690;  1.4039]       12.4
## Hofstad et al.    1.0086 [-0.0002;  2.0174]       10.4
## Frasier et al.   -1.1447 [-1.7143; -0.5751]       13.6
## Azari et al.     -0.2982 [-1.2042;  0.6078]       11.1
## Pagador et al.    0.0585 [-1.3278;  1.4448]        7.9
## 
## Number of studies combined: k = 10
## 
##                          SMD            95%-CI     t p-value
## Random effects model -0.1917 [-0.9642; 0.5809] -0.56  0.5883
## 
## Quantifying heterogeneity:
##  tau^2 = 0.4917 [0.1990; 10.7136]; tau = 0.7012 [0.4461; 3.2732]
##  I^2 = 74.0% [51.2%; 86.1%]; H = 1.96 [1.43; 2.68]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  34.58    9 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolvelocity,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool velocity in Surgery")

#dev.print(pdf, "figures/forest_toolvelocity.pdf", width=8, height=8)

Discussion

Velocity (and related metrics like acceleration) are semi-popular method. Results seem to vary a lot, sometimes novices are faster and sometimes experts are faster. May depend on task?

Tool Acceleration

Tool acceleration measures how much the tool/tools accelerate during the task.

Studies

Load data

df.toolacc <- read_excel('data/surgical_metrics.xlsx', sheet='tool_acceleration')

Print studies

Author	Year	Study	Journal
Azari et al.	2018	Can surgical performance for varying experience be measured from hand motions?	Proceedings of the Human Factors and Ergonomics Society
Frasier et al.	2016	A marker-less technique for measuring kinematics in the operating room	Surgery (United States)
Ebina et al.	2021	Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs	Surgical Endoscopy
Pastewski et al.	2021	Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills	Journal of Surgical Education
Davids et al.	2021	Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation.	World Neurosurgery

Results

Run meta-analysis

m.toolacc <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.toolacc,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Tool acceleration in Surgery")

summary(m.toolacc)

## Review:     Tool acceleration in Surgery
## 
##                      SMD             95%-CI %W(random)
## Azari et al.     -0.3713 [-1.2803;  0.5377]       18.5
## Frasier et al.   -1.0298 [-1.5922; -0.4674]       27.7
## Ebina et al.     -0.7891 [-1.5016; -0.0767]       23.3
## Pastewski et al.  0.1911 [-0.4748;  0.8570]       24.7
## Davids et al.    -0.0233 [-2.0633;  2.0167]        5.8
## 
## Number of studies combined: k = 5
## 
##                          SMD            95%-CI     t p-value
## Random effects model -0.4926 [-1.1602; 0.1750] -2.05  0.1099
## 
## Quantifying heterogeneity:
##  tau^2 = 0.1828 [0.0000; 1.9388]; tau = 0.4275 [0.0000; 1.3924]
##  I^2 = 52.3% [0.0%; 82.5%]; H = 1.45 [1.00; 2.39]
## 
## Test of heterogeneity:
##     Q d.f. p-value
##  8.39    4  0.0782
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.toolacc,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool acceleration in Surgery")

#dev.print(pdf, "figures/forest_toolacceleration.pdf", width=8, height=8)

Discussion

Not many papers that focused on tool accelerations. Jerk (third derivative of position, derivative of acceleration) is much more popular.

Tool Jerk

Jerk is the third derivative of the surgical instruments position, and measures how smooth the movements are.

Studies

Load data

df.jerk <- read_excel('data/surgical_metrics.xlsx', sheet='tool_jerk')

Print studies

Author	Year	Study	Journal
Ghasemloonia et al.	2017	Surgical Skill Assessment Using Motion Quality and Smoothness	Journal of Surgical Education
Hwang et al.	2006	Correlating motor performance with surgical error in laparoscopic cholecystectomy	Surgical Endoscopy and Other Interventional Techniques
Ebina et al.	2021	Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs	Surgical Endoscopy
Azari et al.	2018	Can surgical performance for varying experience be measured from hand motions?	Proceedings of the Human Factors and Ergonomics Society
Davids et al.	2021	Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation.	World Neurosurgery
Oropesa et al.	2013	Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery	Surgical Innovation
Maithel et al	2005	Simulated laparoscopy using a head-mounted display vs traditional video monitor: An assessment of performance and muscle fatigue	Surgical Endoscopy and Other Interventional Techniques
Liang et al.	2018	Motion control skill assessment based on kinematic analysis of robotic end-effector movements	The International Journal of Medical Robotics and Computer Assisted Surgery
Islam et al.	2016	Affordable, web-based surgical skill training and evaluation tool	Journal of Biomedical Informatics
Hofstad et al.	2017	Psychomotor skills assessment by motion analysis in minimally invasive surgery on an animal organ	Minimally Invasive Therapy and Allied Technologies
Shafiel et al.	2017	Motor Skill Evaluation During Robot-Assisted Surgery	Volume 5A: 41st Mechanisms and Robotics Conference
Chmarra et al.	2010	Objective classification of residents based on their psychomotor laparoscopic skills	Surgical Endoscopy and Other Interventional Techniques

Results

Run meta-analysis

m.jerk <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.jerk,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Jerk in Surgery")

summary(m.jerk)

## Review:     Jerk in Surgery
## 
##                         SMD             95%-CI %W(random)
## Ghasemloonia et al.  1.7090 [ 1.1677;  2.2504]        9.6
## Hwang et al.         2.6183 [ 0.1709;  5.0658]        5.0
## Ebina et al.        -0.9365 [-1.6598; -0.2133]        9.3
## Azari et al.        -0.1972 [-0.8043;  0.4098]        9.5
## Davids et al.        0.1307 [-1.9101;  2.1714]        5.9
## Oropesa et al.      -0.9775 [-2.1179;  0.1629]        8.3
## Maithel et al        1.6060 [ 0.7461;  2.4658]        9.0
## Liang et al.         0.1596 [-0.7184;  1.0377]        8.9
## Islam et al.         3.6094 [ 2.4907;  4.7280]        8.3
## Hofstad et al.       1.3201 [-0.1549;  2.7952]        7.4
## Shafiel et al.       0.4174 [ 0.2951;  0.5397]       10.1
## Chmarra et al.       0.8995 [-0.0026;  1.8015]        8.9
## 
## Number of studies combined: k = 12
## 
##                         SMD            95%-CI    t p-value
## Random effects model 0.7932 [-0.0699; 1.6564] 2.02  0.0681
## 
## Quantifying heterogeneity:
##  tau^2 = 1.5140 [0.6120; 5.0878]; tau = 1.2305 [0.7823; 2.2556]
##  I^2 = 87.6% [80.2%; 92.2%]; H = 2.84 [2.25; 3.59]
## 
## Test of heterogeneity:
##      Q d.f.  p-value
##  88.85   11 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.jerk,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Jerk in Surgery")

#dev.print(pdf, "figures/forest_tooljerk.pdf", width=8, height=8)

Discussion

TBD

Tool Force

Tool force is the force the surgeon uses when they e.g. grasp something using the surgical tools.

Studies

Load data

df.force <- read_excel('data/surgical_metrics.xlsx', sheet='tool_force')

Print studies

Author	Year	Study	Journal
Harada et al.	2015	Assessing Microneurosurgical Skill with Medico-Engineering Technology	World Neurosurgery
Prasad et al.	2016	Objective Assessment of Laparoscopic Force and Psychomotor Skills in a Novel Virtual Reality-Based Haptic Simulator	Journal of Surgical Education
Horeman et al.	2014	Assessment of Laparoscopic Skills Based on Force and Motion Parameters	IEEE Transactions on Biomedical Engineering
Trejos et al.	2014	Development of force-based metrics for skills assessment in minimally invasive surgery	Surgical Endoscopy
Woodrow et al.	2007	Training and evaluating spinal surgeons: The development of novel performance measures	Spine
Sugiyama et al.	2018	Forces of Tool-Tissue Interaction to Assess Surgical Skill Level	JAMA Surgery

Results

Run meta-analysis

m.force <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.force,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Force use in Surgery")

summary(m.force)

## Review:     Force use in Surgery
## 
##                    SMD            95%-CI %W(random)
## Harada et al.   0.5357 [-0.0830; 1.1545]       19.3
## Prasad et al.   1.2450 [ 0.6378; 1.8523]       19.3
## Horeman et al.  2.7082 [ 1.5555; 3.8609]       16.3
## Trejos et al.   1.5351 [ 0.2224; 2.8477]       15.3
## Woodrow et al.  3.8205 [ 2.2438; 5.3972]       13.7
## Sugiyama et al. 0.3071 [-0.8880; 1.5022]       16.0
## 
## Number of studies combined: k = 6
## 
##                         SMD           95%-CI    t p-value
## Random effects model 1.5936 [0.2394; 2.9477] 3.03  0.0292
## 
## Quantifying heterogeneity:
##  tau^2 = 1.2415 [0.2843; 10.4203]; tau = 1.1142 [0.5332; 3.2280]
##  I^2 = 79.2% [54.5%; 90.5%]; H = 2.19 [1.48; 3.24]
## 
## Test of heterogeneity:
##      Q d.f. p-value
##  24.02    5  0.0002
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.force,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Force use in Surgery")

#dev.print(pdf, "figures/forest_toolforce.pdf", width=8, height=8)

Discussion

Forces analyzed somewhat commonly, but often not between novices and experts, but within tasks, or tools, or skill groups.

Task-Evoked Pupil Dilation

Pupil size measures cognitive workload, stress, and million other things.

Studies

Load data

df.pupil <- read_excel('data/surgical_metrics.xlsx', sheet='pupil_dilation')

Print studies

Author	Year	Study	Journal
Castner et al.	2020	Pupil diameter differentiates expertise in dental radiography visual search	PLOS ONE
Cabrera-Mino et al.	2019	Task-Evoked Pupillary Responses in Nursing Simulation as an Indicator of Stress and Cognitive Load	Clinical Simulation in Nursing
Bednarik et al.	2018	Pupil Size As an Indicator of Visual-motor Workload and Expertise in Microsurgical Training Tasks	Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications
Gunawardena et al.	2019	Assessing Surgeons’ Skill Level in Laparoscopic Cholecystectomy using Eye Metrics	Eye Tracking Research and Applications Symposium (ETRA)
Dilley et al.	2020	Visual behaviour in robotic surgery—Demonstrating the validity of the simulated environment	International Journal of Medical Robotics and Computer Assisted Surgery
Gao et al.	2018	Quantitative evaluations of the effects of noise on mental workloads based on pupil dilation during laparoscopic surgery	American Surgeon

Results

Run meta-analysis

m.pupil <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.pupil,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Pupil dilation in Surgery")

summary(m.pupil)

## Review:     Pupil dilation in Surgery
## 
##                         SMD             95%-CI %W(random)
## Castner et al.       0.7877 [ 0.6671;  0.9083]       17.6
## Cabrera-Mino et al.  0.8255 [ 0.0502;  1.6009]       16.7
## Bednarik et al.     -2.9791 [-3.5250; -2.4332]       17.1
## Gunawardena et al.   1.5927 [ 0.3701;  2.8152]       15.4
## Dilley et al.       -0.0152 [-0.7136;  0.6833]       16.8
## Gao et al.           1.2184 [ 0.3422;  2.0946]       16.4
## 
## Number of studies combined: k = 6
## 
##                         SMD            95%-CI    t p-value
## Random effects model 0.2082 [-1.5479; 1.9643] 0.30  0.7728
## 
## Quantifying heterogeneity:
##  tau^2 = 2.6787 [0.9588; 16.5443]; tau = 1.6367 [0.9792; 4.0675]
##  I^2 = 97.3% [95.7%; 98.2%]; H = 6.03 [4.85; 7.51]
## 
## Test of heterogeneity:
##       Q d.f.  p-value
##  182.07    5 < 0.0001
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.pupil,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Pupil dilation in Surgery")

#dev.print(pdf, "figures/forest_pupil.pdf", width=8, height=8)

Discussion

Prior research indicates that higher stress/cognitive workload -> larger pupil size. This is seen in most studies. In Bednarik et al. (2018), the effect is reversed. For that study, I picked needle piercing segment (because it was quaranteed to have un-interrupted visual contact from the participant). It can be that experts focused more on this, and had larger cognitive workload and pupil dilations.

Not that many studies that have measured pupil dilations and compared surgical novices and experts directly. Some used measures like ICA or Entropy (not included here). Pupil dilations used in other fields more often.

Tool Grasps

Grasps are the number of times the surgeon had to grasp something using surgical instruments

Studies

Load data

df.grasp <- read_excel('data/surgical_metrics.xlsx', sheet='tool_grasps')

Print studies

Author	Year	Study	Journal
Koskinen et al.	2022	Utilizing Grasp Monitoring to Predict Microsurgical Expertise	Journal of Surgical Research
Cao et al.	1996	Task and Motion Analysis in Endoscopic Surgery	5th Annual Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems

Results

Run meta-analysis

m.grasp <- metagen(TE=g,
                 seTE=SDg,
                 studlab=Author,
                 data=df.grasp,
                 sm="SMD",
                 fixed=FALSE,
                 random=TRUE,
                 method.tau="REML",
                 hakn=TRUE,
                 title="Grasping in Surgery")

summary(m.grasp)

## Review:     Grasping in Surgery
## 
##                    SMD            95%-CI %W(random)
## Koskinen et al. 1.3393 [ 0.7781; 1.9006]       94.2
## Cao et al.      1.0000 [-1.2605; 3.2605]        5.8
## 
## Number of studies combined: k = 2
## 
##                         SMD           95%-CI     t p-value
## Random effects model 1.3196 [0.3112; 2.3280] 16.63  0.0382
## 
## Quantifying heterogeneity:
##  tau^2 = 0; tau = 0; I^2 = 0.0%; H = 1.00
## 
## Test of heterogeneity:
##     Q d.f. p-value
##  0.08    1  0.7752
## 
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Hartung-Knapp adjustment for random effects model

Plot forest

forest.meta(m.grasp,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Grasping in Surgery")

#dev.print(pdf, "figures/forest_grasp.pdf", width=8, height=8)

Discussion

Not many papers that focused on grasps. I have included thus the grasp results also to the “Tool movements” analysis.

How do we measure surgical skill

An informal meta-analysis

Jani Koskinen

Introduction

What is this

Submit results

How results are calculated

Forest plots

Sample size estimation

Task Time

Studies

Results

Discussion

Bimanual Dexterity

Studies

Results

Discussion

Tool Movements

Studies

Results

Discussion

Tool Idle Time

Studies

Results

Discussion

Tool Path Length

Studies

Results

Discussion

Tool Velocity

Studies

Results

Discussion

Tool Acceleration

Studies

Results

Discussion

Tool Jerk

Studies

Results

Discussion

Tool Force

Studies

Results

Discussion

Task-Evoked Pupil Dilation

Studies

Results

Discussion

Tool Grasps

Studies

Results

Discussion